Papers with parameter initialization
Meta Learning and Its Applications to Natural Language Processing (2021.acl-tutorials)
Copied to clipboard
| Challenge: | Meta-learning is a new technique that aims to learn better learning algorithms, including better parameter initialization, optimization strategy, network architecture, distance metrics, and beyond. |
| Approach: | This tutorial introduces Meta-learning approaches and the theory behind them, and then reviews the works of applying this technology to NLP problems. |
| Outcome: | This tutorial will introduce Meta-learning approaches and the theory behind them, and then review the works of applying this technology to NLP problems. |
Decomposed Meta-Learning for Few-Shot Named Entity Recognition (2022.findings-acl)
Copied to clipboard
| Challenge: | Named entity recognition systems aim at recognizing unseen entity types based on a few labeled examples. |
| Approach: | They propose a decomposed meta-learning approach to solve few-shot span detection and few- shot entity typing problems by introducing a model-agnostic meta-loop algorithm. |
| Outcome: | The proposed approach achieves superior performance over prior methods on benchmarks. |
bert2BERT: Towards Reusable Pretrained Language Models (2022.acl-long)
Copied to clipboard
Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, Qun Liu
| Challenge: | Pre-training large language models can be expensive and wasteful. |
| Approach: | They propose a method which can transfer the knowledge of an existing smaller pre-trained model to a large model through parameter initialization and a two-stage learning method to further accelerate the pre-training. |
| Outcome: | The proposed method can transfer the knowledge of an existing smaller pre-trained model to a large model through parameter initialization and significantly improve the pre-training efficiency of the large model. |
Transfer and Multi-Task Learning for Noun–Noun Compound Interpretation (D18-1)
Copied to clipboard
| Challenge: | In computational linguistics, nounnoun compound interpretation is approached as an automatic classification problem. |
| Approach: | They empirically evaluate the utility of transfer and multi-task learning on a challenging semantic classification task. |
| Outcome: | The proposed methods improve the accuracy of a neural classifier and its F1 scores on the less frequent, but more difficult relations. |
Learning from Noisy Labels for Entity-Centric Information Extraction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent information extraction approaches can easily overfit noisy labels and suffer from performance degradation. |
| Approach: | They propose a co-regularization framework for entity-centric information extraction that optimizes neural models with task-specific losses and regularizes them to generate similar predictions based on agreement loss. |
| Outcome: | The proposed framework is optimized with task-specific losses and generates similar predictions based on agreement loss. |
ConsistTL: Modeling Consistency in Transfer Learning for Low-Resource Neural Machine Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing transfer learning methods for low-resource NMT are static, which simply transfer knowledge from a parent model to a child model once via parameter initialization. |
| Approach: | They propose a transfer learning method that can continuously transfer knowledge from the parent model during the training of the child model. |
| Outcome: | The proposed method can transfer knowledge from the parent model to the child model during the training of the child. |
Meta-Learning for Fast Cross-Lingual Adaptation in Dependency Parsing (2022.acl-long)
Copied to clipboard
Anna Langedijk, Verna Dankers, Phillip Lippe, Sander Bos, Bryan Cardenas Guevara, Helen Yannakoudakis, Ekaterina Shutova
| Challenge: | Meta-learning can help overcome resource scarcity in cross-lingual NLP problems . pre-training of models requires large annotated training sets for the task at hand . |
| Approach: | They propose to use meta-learning to train a model to learn a parameter initialization that can adapt quickly to new languages. |
| Outcome: | The proposed model-agnostic meta-learning improves on language transfer and standard supervised learning baselines for unseen, typologically diverse, and low-resource languages in a few-shot learning setup. |
Meta Distant Transfer Learning for Pre-trained Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Notable PLMs are available for text classification tasks, but performance of PLM on downstream tasks may be limited by the availability of training set. |
| Approach: | They propose a meta-learning framework to learn the transferable knowledge across tasks using PLMs. |
| Outcome: | The proposed framework outperforms baselines on seven datasets and is task-agnostic and unbiased. |